A Subtopic Taxonomy-Aware Framework for Diversity Evaluation

To evaluate search result diversification, which is supposed to meet different needs behind a same query, a number of evaluation frameworks are proposed and adopted by benchmarks such as TREC and NTCIR. These frameworks usually do not consider the subtopic taxonomy information. Many previous works on document ranking have shown that different kinds of information needs require different ranking strategies to be satisfied. It is thus necessary to involve subtopic taxonomy in the evaluation framework of search result diversification. In this paper, we propose a novel framework called the Subtopic Taxonomy-Aware (STA) framework to redefine the existing measures. Measures in this new framework take the subtopic taxonomy information into consideration for diversity evaluations. On the other hand, finding the optimal diversified results of many measures is proved as a NPhard problem. We also propose a pruning algorithm which can decrease this problem to a computable search. Experiments based on both the TREC and NTCIR test collections show the effectiveness of our proposed framework.